CRI-O

CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes. It is a lightweight alternative to using Docker as the runtime for kubernetes. It allows Kubernetes to use any OCI-compliant runtime as the container runtime for running pods. Today it supports runc and Kata Containers as the container runtimes but any OCI-conformant runtime can be plugged in principle.

CRI-O supports OCI container images and can pull from any container registry. It is a lightweight alternative to using Docker, Moby or rkt as the runtime for Kubernetes.

What is Kubernetes?

Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google's experience running production workloads at scale with best-of-breed ideas and practices from the community.

What is the CRI and why does Kubernetes need it?

Each container runtime has it own strengths, and many users have asked for Kubernetes to support more runtimes. In the Kubernetes 1.5 release, kubernetes introduced the Container Runtime Interface (CRI) -- a plugin interface which enables kubelet to use a wide variety of container runtimes, without the need to recompile. CRI consists of a protocol buffers and gRPC API, and libraries, with additional specifications and tools under active development.

What is OCI?

The Open Container Initiative (OCI) is a lightweight, open governance structure (project), formed under the auspices of the Linux Foundation, for the express purpose of creating open industry standards around container formats and runtime. The OCI was launched on June 22nd 2015 by Docker, CoreOS and other leaders in the container industry.

The OCI currently contains two specifications: the Runtime Specification (runtime-spec) and the Image Specification (image-spec). The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk. At a high-level an OCI implementation would download an OCI Image then unpack that image into an OCI Runtime filesystem bundle. At this point the OCI Runtime Bundle would be run by an OCI Runtime.

Changes in Kubernetes Workflow Recently

Kubernetes has deprecated the support for Dockershim to optimise the workflow. Now, Kubelet can directly interact with containerd or any other container runtime interface such as CRI-O.

containerd

containerd is a high-level container runtime that came from Docker, and implements the CRI spec. It pulls images from registries, manages them and then hands over to a lower-level runtime, which actually creates and runs the container processes.

containerd was separated out of the Docker project, to make Docker more modular.

So Docker uses containerd internally itself. When you install Docker, it will also install containerd.

containerd implements the Kubernetes Container Runtime Interface (CRI), via its cri plugin.

CRI-O

CRI-O is a high-level container runtime which implements the Container Runtime Interface (CRI). It’s an alternative to containerd. It pulls container images from registries, manages them on disk, and launches a lower-level runtime to run container processes.

It was specifically created from the ground up as a container runtime for Kubernetes as Kubernetes has deprecated it's support for dockershim. It provides the ability to start, stop and restart containers, just like containerd.

Contributors

CRI-O is developed by maintainers and contributors from these companies and others. It is a community-driven, open source project. Feedback, users, and of course, contributors, are always welcome via the cri-o/cri-o GitHub project.

Architecture

The architectural components are as follows:

• Kubernetes contacts the kubelet to launch a pod.

  - Pods are a kubernetes concept consisting of one or more containers sharing the same IPC, NET and PID `namespaces` and living in the same cgroup.

• The kubelet forwards the request to the CRI-O daemon VIA kubernetes CRI (Container runtime interface) to launch the new POD.
• CRI-O uses the containers/image library to pull the image from a container registry.
• The downloaded image is unpacked into the container’s root filesystems, stored in COW file systems, using containers/storage library.
• After the rootfs has been created for the container, CRI-O generates an OCI runtime specification json file describing how to run the container using the OCI Generate tools.
• CRI-O then launches an OCI Compatible Runtime using the specification to run the container proceses. The default OCI Runtime is runc.
• Each container is monitored by a separate conmon process. The conmon process holds the pty of the PID1 of the container process. It handles logging for the container and records the exit code for the container process.
• Networking for the pod is setup through use of , so any CNI plugin can be used with CRI-O.

Components

CRI-O is made up of several components that are found in different GitHub repositories.

• OCI compatible runtime
• containers/storage
• containers/image
• networking (CNI)
• container monitoring (conmon)
• Security is provided by several core Linux capabilities

OCI compatible runtimes

CRI-O supports any . They test with runc and Clear Containers today.

Storage

The library is used for managing layers and creating root file-systems for the containers in a pod: Overlayfs, devicemapper, AUFS and btrfs are implemented, with Overlayfs as the default driver.

Container images

The library is used for pulling images from registries. Currently, it supports Docker as well as . It also passes all Docker and Kubernetes tests.

Networking

The Container Network Interface is used for setting up networking for the pods. Various CNI plugins such as Flannel, Weave and OpenShift-SDN have been tested with CRI-O and are working as expected.

Monitoring

is a utility within CRI-O that is used to monitor the containers, handle logging from the container process, serve attach clients and detects Out Of Memory (OOM) situations.

Security

Container security separation policies are provided by a series of tools including SELinux, Capabilities, seccomp, and other security separation policies as specified in the OCI Specification.

CRI-O Installation Instructions

This guide will walk you through the installation of CRI-O, an Open Container Initiative-based implementation of the Kubernetes Container Runtime Interface. It is assumed you are running a Linux machine.

Table of Contents:

• Install packaged versions of CRI-O
  • Supported versions
  • Installation Instructions
    • openSUSE
    • Fedora 31 or later
    • Other yum based operating systems
    • APT based operating systems
• Build and install CRI-O from source
  • Runtime dependencies
  • Build and Run Dependencies
    • Fedora - RHEL 7 - CentOS
    • RHEL 8
    • Debian - Raspbian - Ubuntu
  • Get Source Code
  • Build
    • Install with Ansible
    • Build Tags
  • Static builds
    • Creating a release archive
  • Download conmon
  • Setup CNI networking
  • CRI-O configuration
    • Validate registries in registries.conf
    • Recommended - Use systemd cgroups.
    • Optional - Modify verbosity of logs
    • Optional - Modify capabilities and sysctls
  • Starting CRI-O
  • Using CRI-O

Install packaged versions of CRI-O

CRI-O builds for native package managers using openSUSE's OBS

Supported versions

CRI-O follows the Kubernetes support cycle of three minor releases. CRI-O also attempts to package for the following operating systems:

Fedora 31+
openSUSE
CentOS 8
CentOS 8 Stream
CentOS 7
Debian Unstable
Debian Testing
Debian 10
Rasbian 10
xUbuntu 20.10
xUbuntu 20.04
xUbuntu 18.04

To install, choose a supported version for your operating system, and export it as a variable, like so: export VERSION=1.19

We also save releases as subprojects. If you'd, for instance, like to use 1.18.3 you can set export VERSION=1.19:1.19.1

Packaging for CRI-O is done best-effort, and is largely driven by requests. If there's a version or operating system that is missing, please open an issue.

Installation Instructions

openSUSE:

sudo zypper install cri-o

Fedora 31 or later

sudo dnf module enable cri-o:$VERSION
sudo dnf install cri-o

For Fedora, we only support setting minor versions. i.e: VERSION=1.18, and do not support pinning patch versions: VERSION=1.18.3

Other yum based operating systems

To install on the following operating systems, set the environment variable $OS as the appropriate field in the following table:

Operating system	$OS
Centos 8	CentOS_8
Centos 8 Stream	CentOS_8_Stream
Centos 7	CentOS_7

And then run the following as root:

curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
yum install cri-o

APT based operating systems

Note: this tutorial assumes you have curl and gnupg installed

To install on the following operating systems, set the environment variable $OS as the appropriate field in the following table: | Operating system | $OS | | ---------------- | ----------------- | | Debian Unstable | Debian_Unstable | | Debian Testing | Debian_Testing | | Raspberry Pi OS | Raspbian_10 | | Ubuntu 20.04 | xUbuntu_20.04 | | Ubuntu 19.10 | xUbuntu_19.10 | | Ubuntu 19.04 | xUbuntu_19.04 | | Ubuntu 18.04 | xUbuntu_18.04 |

If installing cri-o-runc (recommended), you'll need to install libseccomp >= 2.4.1. This is not available in distros based on Debian buster or below, so buster backports will need to be enabled:

echo 'deb http://deb.debian.org/debian buster-backports main' > /etc/apt/sources.list.d/backports.list
apt update
apt install -y libseccomp2 || apt update -y libseccomp2

And then run the following as root:

echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

apt-get update
apt-get install cri-o cri-o-runc

Note: We include cri-o-runc because Ubuntu and Debian include their own packaged version of runc. While this version should work with CRI-O, keeping the packaged versions of CRI-O and runc in sync ensures they work together. If you'd like to use the distribution's runc, you'll have to add the file:

[crio.runtime.runtimes.runc]
runtime_path = ""
runtime_type = "oci"
runtime_root = "/run/runc"

to /etc/crio/crio.conf.d/

Build and install CRI-O from source

Runtime dependencies

• runc, Clear Containers runtime, or any other OCI compatible runtime
• iproute
• iptables

Latest version of runc is expected to be installed on the system. It is picked up as the default runtime by CRI-O.

Build and Run Dependencies

Fedora - RHEL 7 - CentOS

Required

Fedora, RHEL 7, CentOS and related distributions:

yum install -y \
  containers-common \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  pkgconfig \
  make \
  runc

Please note:

• CentOS 8 (or higher): pkgconfig package is replaced by pkgconf-pkg-config
• By default btrfs is not enabled. To add the btrfs support, install the following package: btrfs-progs-devel
• It is possible the distribution packaged version of runc is out of date. If you'd like to get the latest and greatest runc, consider using the one found in https://build.opensuse.org/project/show/devel:kubic:libcontainers:stable

RHEL 8

RHEL 8 distributions:\ Make sure you are subscribed to the following repositories: \ BaseOS/x86_64 \ Appstream/x86_64 CodeReady Linux Builder for x86_64

subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms
subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms
subscription-manager repos --enable=codeready-builder-for-rhel-8-x86_64-rpms

Follow the guide below to subscribe to the repositories if not already subscribed:\ https://access.redhat.com/solutions/265523

This requires go version 1.12 or greater:

yum module -y install go-toolset

yum install -y \
  containers-common \
  device-mapper-devel \
  git \
  make \
  glib2-devel \
  glibc-devel \
  glibc-static \
  runc \

Here is a link on how to install a source rpm on RHEL: \ https://www.itechlounge.net/2012/12/linux-how-to-install-source-rpm-on-rhelcentos/

Dependency: gpgme-devel \ Link: http://download.eng.bos.redhat.com/brewroot/packages/gpgme/1.10.0/6.el8/x86_64/

Dependency: go-md2man \ Command:

go get github.com/cpuguy83/go-md2man

The following dependencies:

  libassuan \
  libassuan-devel \
  libgpg-error \
  libseccomp \
  libselinux \
  pkgconf-pkg-config \

Debian - Raspbian - Ubuntu

On Debian, Raspbian and Ubuntu distributions, enable the Kubic project repositories and install the following packages:

apt-get update -qq && apt-get install -y \
  btrfs-tools \
  containers-common \
  git \
  golang-go \
  libassuan-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libc6-dev \
  libgpgme11-dev \
  libgpg-error-dev \
  libseccomp-dev \
  libsystemd-dev \
  libselinux1-dev \
  pkg-config \
  go-md2man \
  cri-o-runc \
  libudev-dev \
  software-properties-common \
  gcc \
  make

Caveats and Notes:

If using an older release or a long-term support release, be careful to double-check that the version of runc is new enough (running runc --version should produce spec: 1.0.0), or else build your own.

Be careful to double-check that the version of golang is new enough, version 1.12.x or higher is required. If needed, newer golang versions are available at the official download website.

Get Source Code

Clone the source code using:

git clone https://github.com/cri-o/cri-o # or your fork
cd cri-o

Make sure your CRI-O and kubernetes versions are of matching major versions. For instance, if you want to be compatible with the latest kubernetes release, you'll need to use the latest tagged release of CRI-O on branch release-1.18.

Build

To install with default buildtags using seccomp, use:

make
sudo make install

Otherwise, if you do not want to build CRI-O with seccomp support you can add BUILDTAGS="" when running make.

make BUILDTAGS=""
sudo make install

Install with Ansible

An Ansible Role is also available to automate the above steps:

sudo su -
mkdir -p ~/.ansible/roles
cd ~/.ansible/roles
git clone https://github.com/alvistack/ansible-role-cri_o.git cri_o
cd ~/.ansible/roles/cri_o
pip3 install --upgrade --ignore-installed --requirement requirements.txt
molecule converge
molecule verify

Build Tags

CRI-O supports optional build tags for compiling support of various features. To add build tags to the make option the BUILDTAGS variable must be set.

make BUILDTAGS='seccomp apparmor'

Build Tag	Feature	Dependency
seccomp	syscall filtering	libseccomp
selinux	selinux process and mount labeling	libselinux
apparmor	apparmor profile support

CRI-O manages images with containers/image, which uses the following buildtags.

Build Tag	Feature	Dependency
containers_image_openpgp	use native golang pgp instead of cgo
containers_image_ostree_stub	disable use of ostree as an image transport

CRI-O also uses containers/storage for managing container storage.

Build Tag	Feature	Dependency
exclude_graphdriver_btrfs	exclude btrfs as a storage option
btrfs_noversion	for building btrfs version < 3.16.1	btrfs
exclude_graphdriver_devicemapper	exclude devicemapper as a storage option
libdm_no_deferred_remove	don't compile deferred remove with devicemapper	devicemapper
exclude_graphdriver_overlay	exclude overlay as a storage option
ostree	build storage using ostree	ostree

Static builds

It is possible to build a statically linked binary of CRI-O by using the officially provided nix package and the derivation of it within this repository. The builds are completely reproducible and will create a x86_64/amd64 or aarch64/arm64 stripped ELF binary for glibc. These binaries are integration tested as well and support the following features:

• apparmor
• btrfs
• device mapper
• gpgme
• seccomp
• selinux

To build the binaries locally either install the nix package manager or use the make build-static target which relies on the nixos/nix container image.

The overall build process can take a tremendous amount of CPU time depending on the hardware. The resulting binaries should now be available within:

• bin/static/crio

To build the binaries without any prepared container and via the already installed nix package manager, simply run the following command from the root directory of this repository:

nix build -f nix

The resulting binaries should be now available in result/bin. To build the arm variant of the binaries, just run:

nix build -f nix/default-arm64.nix

Creating a release archive

A release bundle consists of all static binaries, the man pages and configuration files like 00-default.conf. The release-bundle target can be used to build a new release archive within the current repository:

make release-bundle
…
Created ./bundle/cri-o.amd64.v1.20.0.tar.gz

Download conmon

conmon is a per-container daemon that CRI-O uses to monitor container logs and exit information. conmon needs to be downloaded with CRI-O.

running:

git clone https://github.com/containers/conmon
cd conmon
make
sudo make install

will download conmon to your $PATH.

Setup CNI networking

A proper description of setting up CNI networking is given in the contrib/cni README. But the gist is that you need to have some basic network configurations enabled and CNI plugins installed on your system.

CRI-O configuration

If you are installing for the first time, generate and install configuration files with:

sudo make install.config

Validate registries in registries.conf

Edit /etc/containers/registries.conf and verify that the registries option has valid values in it. For example:

[registries.search]
registries = ['registry.access.redhat.com', 'registry.fedoraproject.org', 'quay.io', 'docker.io']

[registries.insecure]
registries = []

[registries.block]
registries = []

For more information about this file see registries.conf(5).

Optional - Modify verbosity of logs

Users can modify the log_level by specifying an overwrite like /etc/crio/crio.conf.d/01-log-level.conf to change the verbosity of the logs. Options are fatal, panic, error, warn, info (default), debug and trace.

[crio.runtime]
log_level = "info"

Optional - Modify capabilities and sysctls

By default, CRI-O uses the following capabilities:

default_capabilities = [
    "CHOWN",
    "DAC_OVERRIDE",
    "FSETID",
    "FOWNER",
    "SETGID",
    "SETUID",
    "SETPCAP",
    "NET_BIND_SERVICE",
    "KILL",
]

and no sysctls

default_sysctls = [
]

Users can change either default by adding overwrites to /etc/crio/crio.conf.d.

Starting CRI-O

Running make install will download CRI-O into the folder

/usr/local/bin/crio

You can run it manually there, or you can set up a systemd unit file with:

sudo make install.systemd

And let systemd take care of running CRI-O:

sudo systemctl daemon-reload
sudo systemctl enable crio
sudo systemctl start crio

Using CRI-O

• Follow this tutorial to quickly get started running simple pods and containers.
• To run a full cluster, see the instructions.
• To run with kubeadm, see kubeadm instructions.